The first structure on this strategic site overlooking the River Nore was a wooden tower built in 1172 by the Anglo-Norman conqueror of Ireland, Richard de Clare, also known as Strongbow. Twenty years later his son in law William Marshall erected a stone castle with four towers, three of which are part of the current structure.
The powerful Butler family bought the castle in 1391, and their descendents continued to live there until 1935. Maintaining the castle became such an enormous expense and most of the furnishings were sold at an auction. The city bought the castle in 1967.